Chapter 1: Exploratory Data Analysis

Introduction

John Tukey is often regarded as one of the founding drivers for modern data science, incorporating statistical inference along with basic summary statistics to visualize and understand data.

# Load libraries
suppressPackageStartupMessages(library(tidyverse))
library(curl)
library(ggtext)
library(scales)
library(patchwork)
library(kableExtra)
library(matrixStats)

# Set ggplot2 theme
my_theme <- theme(
  panel.grid = element_blank(), 
  panel.background = element_rect(fill = "white", color = "black"),
  
  # Set axis and title themes
  plot.title = element_markdown()
)

Elements of Structured Data

The two main types of structured data (i.e., data in a table similar to an Excel spreadsheet or something similar) are numeric and categorical:

• Numeric:

  • Continuous: any real number (e.g., velocity, flow rates)

  • Discrete: counts (e.g., number of children per household)

• Categorical: fixed set of values (e.g., car models, species of fish)

  • Binary: true/false, 0/1, yes/no values

  • Ordinal: fixed set of values with specified order to denote rating (e.g., 1-5)

Estimates of Location

The mean describes the average value, taken by summing all of the values and dividing by the total number of records (i.e., total number of observations).

# Take the mean
mean(random_data)

[1] 9.940272

If some cases, the mean may not be an appropriate summary statistic for the dataset. For example, the presence of outliers could skew the mean and lead to eroneous conclusions. In these cases, it may be appropriate to use the trimmed mean, which is calculated by dropping a fixed set of of sorted values from each end of the distribution. The weighted mean is another variation that may be used in situations where data from some features are more important than others, due to intrinsic variability in some features, or due to sampling design that wasn’t representative of the entire population.

We can calculate the trimmed mean using the mean() function and declaring the trim argument to remove a fraction of observations from each end of the distribution before the mean is calculated.

mean(random_data, trim = 0.5)

[1] 9.904327

We can calculate the weighted mean using the weighted.mean() function, which takes two arguments: x to define the object containing the values whose weighted mean we want to calculate; and w, to define an object of the same length of x giving the weights of each element of x. In the example below, we’ll use data from Siegel 1994 (also available in the help documentation) to calculate the weighted GPA, which is the GPA that considers the grade the student received in a course as well as the difficulty of the course.

# GPA from Siegel 1994- example from the help documentation
wt <- c(5, 5, 4, 1)/15
x <- c(3.7, 3.3, 3.5, 2.8)

# Calculate the weighted mean pf GPA for each quarter, weighted by the course load
weighted.mean(x, wt)

[1] 3.453333

Median and Robust Estimates

The median is less sensitive to the data, meaning that it is robust to outliers present in a dataset. In cases with an odd number of records, the median is the middle number; in cases with an even number of records, the median is the average between the two middle numbers that divide the data into lower and upper halves.

Anomaly Detection

Although sometimes, outliers (extreme data points far from the mean) are mistakes made from data entry, there would be something interesting going on to warrant further investigation. In general, though, either the median or trimmed mean are used, trimming 10% of the lowest and highest values from the sorted distribution before calculating the mean, in the latter case.

Example: Location Estimates of Population and Murder Rates

In this example, we’ll explore the murder rates across the country in the states.csv file.

We can compute various summary statistics on this dataset to get an overview of murder rates across the United States.

Population Size
Mean	Mean (trimmed)	Median
6,162,876	4,783,697	4,436,370

We can also calculate the weighted mean and median, which will take into account the population size for each state. The key takeaway here is that both estimates are relatively similar.

Weighted statistics
Mean	Median
4.445834	4.4

Estimates of Variability

Variability describes the spread of data and distinguishing it from real variability. There are a few terms to review, including deviations (differences between the observed and estimated values, like the mean, for example), variance (sum of squared deviations from the mean, divided by the sample size minus one), and the standard deviation (the square root of the variance). There are many other metrics used to describe variability, which will be demonstrated using examples below.

One metric to describe variability is the mean absolute deviation, or the average of the absolute differences between the observed and estimated values. More commonly used metrics are the variance (average of the squared deviations) and the standard deviation. The latter metric is preferred over the variance because it is in the same scale as the original data.

Often, the variance and standard deviation formulas use n-1 in their denominators to generate unbiased estimates. When using just n, the sample estimates of the variance and standard deviations will be biased and underestimate the true population parameters, because of the constraint that the standard deviation requires calculating the sample mean.

Variance and standard deviation are both sensitive to outliers, but another metric, the median absolute deviation from the median (MAD), is not (similar to the median).

Estimates Based on Percentiles

The percentile is the value at which a given percentage of the data are a particular value, or less. For example, if we took a sequence of numbers from 1 to 1,000 and piped them into the quantile() function, we’d see that the 25th percentile is 250.75. In other words, 25% of the values from 1 to 1,000 are 250.75 or less. The IQR() function will yield the interquartile range, or the essentially where the middle 50% of the data lie.

Below, we’ll explore the states dataset using the estimates of variability.

From this plot, it’s obvious that the standard deviation (SD) is about 1.7 times greater than the MAD, because it is sensitive to outliers.

Exploring the Data Distribution

There are several additional approaches to visualizing distributions, including histograms and density plots. These two visualizations are similar, except the histograms are based on binning observations into discrete bins, while the latter involves kernel density estimation (KDE) to approximate the probability distribution. KDE is a non-parametric approach to approximating the probability distribution because sometimes data don’t conform to the standard probability distributions (e.g., if data are bimodal, or multimodal). In parametric approaches, we can describe the probability distribution using parameters. For example, if data are normally distributed, we describe the probability distribution using two parameters: the mean and standard deviation.

To describe the middle 50% of the data for various fields, or the range between the 25th and 75th percentile, we can use a boxplot. The data points outside of 1.5 times the IQR are defined as outliers (also shown). Note in the boxplots, the function call should be ggplot(aes("", Population)), or else there will be erroneous data on the x-axis.

Statistical Moments

There are four moments of a distribution:

1. Location (referring to mean, median, mode)

2. Variability (e.g., standard deviation, variance, MAD)

3. Skewness (are data skewed to large or small values)

4. Kurtosis (describes the tendency for data to have extreme values)

Exploring Binary and Categorical Data

In the next section, we’ll explore categorical data using the dfw_airline.csv dataset, which contains data on the number of delayed flights per year from Dallas/Fort Worth airport due to various reasons: carrier issues, air traffic control problems, weather, security, or issues with inbound flights.

Carrier	ATC	Weather	Security	Inbound
64263.16	84856.5	11235.42	343.15	118427.8

These data can also be represented using a bar plot.

Correlation

The correlation describes the relationship between two variables on a continuous -1 to 1 scale. It is calculated by taking the deviations from the mean for one variable, multiplied by the deviations from the mean for the second variable, and then dividing by the product of the standard deviations. In this section, we’ll explore the sp500_px.csv dataset, which contains 5,647 records of telecommunication stocks from 517 different companies over time. To create the plot in the textbook, we’ll also import the sp500_sym.csv dataset, which contains data on the company symbols and sectors. We’ll join both of these together, then filter the final dataset to only include those records for major exchange-tranded funds (ETFs) between July 2012 through June 2015.

Now, we’ll calculate the correlation of daily returns between all companies.

etf_data %>% 
  select(symbol, daily_returns) %>% 
  mutate(rownumber = row_number()) %>% 
  pivot_wider(id_cols = rownumber, 
              names_from = "symbol", 
              values_from = "daily_returns")